Skip to content

fix(fleet): retire a start's capacity wait on the next attempt - #160

Merged
outofcoffee merged 1 commit into
mainfrom
worktree-dashboard-start-no-capacity
Sep 3, 2026
Merged

fix(fleet): retire a start's capacity wait on the next attempt#160
outofcoffee merged 1 commit into
mainfrom
worktree-dashboard-start-no-capacity

Conversation

@outofcoffee

@outofcoffee outofcoffee commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

A remote node's dashboard tile no longer reports a capacity wait for the rest of a start that has since found capacity and booted.

Summary

  • Wire remote.Start's onState callback through in remoteNode.StartWithProgress. It reports StateInFlight whenever a fresh attempt goes out, retiring whatever the previous attempt said — the mechanism the dashboard needed and never received, because the fleet node passed nil for it.
  • Guard a nil progress callback in StartWithProgress, so a caller wanting no lines says so explicitly rather than depending on which paths a given start happens to take.
  • Show elapsed time beside the in-flight verb on the tile, recomputed on each repaint rather than fixed when a status line arrives.
  • Record both behaviours in openspec/specs/fleet-client/spec.md: a status line must describe the node's situation now, and the tile's timings must be computed as it is drawn.

Implementation details

internal/remote.Start reports through two channels. progress func(string) writes a line only immediately before a wait; onState func(string) reports the state of every poll, plus StateInFlight when a new attempt is issued. StateInFlight exists precisely for this case — its comment reads "an in-flight attempt supersedes an earlier no-capacity report", and internal/remote/remote_test.go covers it. internal/fleet/remote_node.go passed nil for it, so only spinloop remote start benefited.

That left the tile holding just the progress lines, and the sequence is:

  1. a 503 no-capacity writes instance no-capacity; retrying in 120s, which the tile stores
  2. capacity frees up, and the next attempt gets a 200 that blocks until the model is serving — so no further line for minutes
  3. the 60-second metrics round returns running, and is drawn below the stale line

The CLI never showed this because it passes progress.setState and repaints over the stale text on a 30-second heartbeat. The dashboard had neither, which is why the tile could show a capacity wait and a running instance at once.

The elapsed counter is deliberately derived from the board's clock at draw time rather than baked into a line when it arrives — a start's own lines can legitimately stand unchanged for minutes, so a moving number is what separates a tile that is waiting from one that is wedged. dashNow is a package variable so the tile's output stays byte-stable under test.

Wiring onState up also turned a latent nil dereference into a real one: StartWithProgress(ctx, nil) was previously safe only because progress was never called on the path an existing test took. An existing test caught it as a panic, hence the guard.

This is the narrow fix. It leaves the wider fragility in place — the tile still renders a transcript line as a status, nothing carries a read timestamp (so a refresh that lands late can repaint stale data), and remote nodes stay on the 60-second cadence during a start. An openspec proposal for that rebuild follows separately.

@outofcoffee outofcoffee added the bug Something isn't working label Sep 3, 2026
@outofcoffee outofcoffee changed the title fix(fleet): retire a start's capacity wait once the next attempt goes out fix(fleet): retire a start's capacity wait on the next attempt Sep 3, 2026
@outofcoffee outofcoffee added the go Pull requests that update go code label Sep 3, 2026
… out

A remote start refused for capacity reports "instance no-capacity; retrying
in 120s" — true until the next attempt is issued. The attempt that finds
capacity then holds one request for the whole boot and reports nothing more,
so the dashboard tile, which shows the latest line as the node's current
situation, went on reporting a capacity wait for minutes, beside its own
refreshes reporting the node running.

remote.Start already has the fix: it calls onState with StateInFlight when a
fresh attempt goes out, exactly so an observer can retire the previous
attempt's verdict. The fleet node passed nil for it, so only the CLI got the
benefit. Wire it through, and turn it into a line the tile can show.

Guard a nil progress callback while here: remote.Start writes its lines
unconditionally, so a caller passing nil was relying on which paths the start
happened to take.

Add the elapsed time beside the in-flight verb, recomputed on each repaint.
A start's lines can legitimately stand unchanged for minutes, so a moving
number is what separates a tile that is waiting from one that is wedged.
@outofcoffee
outofcoffee force-pushed the worktree-dashboard-start-no-capacity branch from 157d8fd to d610609 Compare September 3, 2026 21:44
@outofcoffee
outofcoffee merged commit 03718af into main Sep 3, 2026
2 checks passed
@outofcoffee
outofcoffee deleted the worktree-dashboard-start-no-capacity branch September 3, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working go Pull requests that update go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant